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and the subject of change. In fact, the requirement for capacity is present 
at all levels of the system in systemic reform. In educational change, four 
types of capacity are generally considered: (1) human capacity; (2) 

organizational capacity; (3) structural capacity; and (4) material capacity. 
Because there are so many types of capacity, evaluators may find a general 
organizing structure for looking at capacity useful before they set out to 
identify specific questions and indicators for evaluating capacity. A matrix 
is provided to allow evaluators to focus on the different types of capacity 
as "drivers" of targeted studies designed to match the more immediate 
strategies and goals of the reform. Examining capacity is one of the ways 
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Introduction 



Systemic reform is a newly developing approach to educational improvement that is generating new 
language for the practice and study of educational reform. "Capacity" has become part of this evolving 
lexicon. While not entirely new to the language of educational change, "capacity" has adopted more rich, 
complex meanings in the systemic reform arena. Spillane and Thompson write that current reforms are 
"even more dependent on local capacity than were past waves of reform" (Spillane & Thompson, 1997) 
suggesting that capacity has matured from a mere descriptor of a person or program to become one of the 
core attributes of the systemic reform process. In its evolution, capacity has opened new doors and 
created new challenges for evaluators. 

The Importance of Capacity in the Context of Systemic Chang e 

Past reforms in mathematics and science have most often targeted programmatic areas of the education 
system such as changes in curriculum, teaching practices, or assessments. Evaluators then determine the 
value of these reforms by assessing the extent to which the target curricula, teaching practices or 
assessments are used and understood. Systemic reforms, one could argue, are different. In addition to 
such programmatic goals, systemic efforts focus on long-term time lines, promote reflection and strategic 
improvement, and hold the expectation that change will be continuous. Therefore, they can be described 
as having two domains of goals - programmatic goals (like conventional reforms) and goals targeted to 
building capacity for continuous change and improvement (Century, 1997). 

This notion of working toward continuous change is not unique to education. In writing about business 
systems McAdams suggests that an organization with capacity is one that is "agile". By agility, he means 
that it can enrich the customer, cooperate to enhance competitiveness, organize to master change and 
uncertainty and leverage the impact of people and information (McAdams, 1997). It is not difficult to see 
how such "agility" might be desirable for an education system undergoing a mathematics or science 
systemic reform process. 

Evaluators, then, face a new challenge in evaluating systemic reform. Given the dual domains of goals, a 
comprehensive evaluation of a mathematics or science systemic reform should include more than 
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recognizing the current status of a program and making judgments on that program's worth. It also needs 
to identify and assess the capacity of the program leaders, the participants, and the system itself for 
maintaining the program into the future and for facilitating steady, continued improvement. 



Capacity is also particularly pertinent in systemic reform because in it, the system can be both the initiator 
and the subject of change. The notion that "the changer is the changee" (St. John, 1996) meaning, that those 
who are responsible for initiating and implementing the change are also among the subjects of the change, 
is a key idea in systemic reform. Such reforms depart from a more conventional model of having outside 
"experts" (e.g. universities or professional organizations) facilitate changes in others (schools, districts and 
states). Although systems can purchase technical assistance from external sources, ideally, leaders of 
reforms within the system ultimately must have the capacity to make the reform happen. Furthermore, 
the system itself must have the capacity to embrace the reform and support its continuous development. 
This raises significant challenges for the evaluator in considering whether the system is capable of, or 
inclined to change itself and in turn, how successful the systemic reform is likely to be. 



The requirement for capacity is present at all levels of the system in systemic reform. Linda Darling 
Hammond writes that systemic reform is a "changed mission" for education that requires a "new model 
for school reform, one in which policy makers shift their efforts from designing controls intended to direct 
the system, to developing the capacity of schools and teachers to be responsible for student learning and 
responsive to student and community needs, interests and concerns." She points out that instead of 
standardizing practice, school reforms must "focus on building the capacity of schools and teachers to 
undertake tasks they have never before been called upon to accomplish" (Darling-Hammond, 1993). 

The challenge for evaluators, then, is multifaceted. They must turn their attention to judging the capacities 
of the individuals leading and participating in the reform, and they must judge the capacity of the system 
itself. Because evaluators need to look at such a wide range of contexts for capacity it is important to first 
establish a clear understanding of what capacity is and how to describe and define it. Following is an 
organizational framework for defining capacity that can help organize a capacity study as part of a 
comprehensive systemic reform evaluation. 
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Current Understandings and Definitions of Capacity 

In a most general sense, capacity refers to an entity's (an individual, group of individuals, system, etc.) 
ability to achieve the goals of a reform. In traditional improvement programs, this may simply refer to a 
university's ability to carry out a professional development program for teachers, or a state's ability to 
establish a new policy. In systemic reform, however, meanings of "capacity" have proliferated to 
encompass the various components and proponents of the reform as well as newly emerging goals and 
roles. Understanding these multiple definitions is necessary groundwork for fully understanding and 
making use of strategies for evaluating capacity. 

The literature on educational change holds a range of descriptions of capacity, some of which reflect 
increased attention to the systemic context. Spillane and Thompson, for example, conducted a study 
which identified three components of a system's (a school district) capacity for change: human capital, 
social capital, and financial resources. By their definition, human capital refers to the commitment, 
disposition and knowledge of local reformers; social capital refers to the relations among individuals in a 
group or organization both with each other and those outside the district and financial resources refers to 
those resources allocated to staffing, time and materials (Spillane & Thompson, 1997). Their work 
illustrates how shifting from a traditional reform effort to a systemic one that is district-wide and long- 
lasting calls for a commensurate shift from a simple to a multi-faceted definition of capacity. This paper 
draws from this and other descriptions of capacity by organizing its multiple definitions into four broad 
categories: human capacity, organizational capacity, structural capacity, and material capacity. 

1) Human Capacity 

Human capacity is comprised of two parts: intellectual proficiency and will. Intellectual proficiency 
encompasses the knowledge, expertise and understanding that the people leading and engaged in the 
mathematics or science systemic reform must hold. Will is comprised of the interest, patience and 
persistence necessary to implement a mathematics or science systemic reform. Ideally, every group with 
an interest or stake in improving the mathematics or science program in a system must have sufficient 



human capacity to carry out its requisite role. In systemic reform, such groups are many and may include: 
teachers, school administrators, central office administrators, university faculty, business owners, and 
students. 

Evaluators, then, need to identify and assess the characteristics that indicate high human capacity in the 
individuals in each group. It is unlikely that any systemic reform effort will have a full complement of 
high capacity individuals at the outset; some groups within the reform may have few high capacity 
individuals, while others may have many. The key is that the high capacity individuals are at strategic 
points in the reform, and that there is evidence that human capacity will continue to grow. 

2) Organizational Capacity 

Organizational capacity refers to the interaction, collaboration, and communication among individuals in 
the system. Assume that an education system (whether a state or a district) is comprised of individuals in 
organizational structures. One can not assume that the overall organizational capacity is equal to the sum 
of the human capacities of the individuals that make it up. Rather, the interactions individuals have with 
one another shape a culture; a tone for collaboration and a means of communication that together 
comprise their organizational capacity. Some may under-emphasize the importance of giving attention to 
organizational capacity, but according to McAdams, although change is often viewed as a rational 
process, "in reality organizations change only when the people in them are willing and able to do so... one 
can say that an organization needs to be recultured before it can be restructured." (McAdams, 1997) 

3) Structural Capacity 

Structural capacity is defined as the elements of the system which exist independent of the human beings 
who may use or change those elements in order to function in the system. Such elements include policies, 
procedures and formalized practices such as: curriculum frameworks; professional development program 
designs; teacher certification policies; hiring practices; curriculum adoption procedures; formal 
procedures for collaboration amongst central office personnel; and formal partnerships between 
universities and the school system. 



In looking at these characteristics of the system, the evaluator needs to consider more than the simple 
presence of a structure in the system. Evaluators also need to assess the quality of each element and its 
likelihood for sustainability. Even if highly visible, a structural element that is of low quality, or that is 
difficult to maintain should not be considered as favorable for capacity. It may represent a sort of 
"temporary" capacity, but can not warrant serious consideration in the long-term context of systemic 
reform. 

4) Material Capacity 

Material capacity is comprised of the fiscal resources and other material supports available to the reform 
and to the mathematics or science program in the system. Financial resources include funds external to 
the system's (e.g. district or state) budget (if there are any) as well as those funds allocated internally to 
support the reform and the mathematics or science program. Other material supports can take many 
forms and include storage and meeting space, instructional materials, supplemental curriculum resources, 
transportation, meeting supports, and technological capability. 

Evaluators need to identify the existing resources and supports that come from within, and from outside 
the system and consider the extent to which those supports are likely to continue. Evaluating financial 
capacity may include a close look at reallocation of dollars within the system or identifying potential for 
obtaining resources in the future. 

Interactions Between, and Interdependence of the Various Capacities 

Spillane and Thompson note that the various capacities of a system are intertwined and that growth in 
one area depends closely on growth in another (Spillane & Thompson, 1997). The interdependence of 
these aspects of capacity, it seems, reflects the interdependence of the components of a reform effort as a 
whole. Understanding these relationships is key for evaluating and drawing accurate conclusions about 
capacity. 

Consider human and organizational capacity. One might conclude that a system with high human 
capacity is likely to have high organizational capacity as well. However, this may not necessarily be the 
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case; a system may have high capacity individuals who fail to communicate or collaborate. Without a 
complementary organizational capacity, the high level of human capacity isn't completely realized in the 
system as a whole. 

Likewise, material capacity and structural capacity are interdependent. An evaluator may identify a well- 
targeted policy or well-designed professional development plan, and then find that there are insufficient 
resources to carry it out. Or, perhaps there may be sufficient resources but the human capacity is 
inadequate to carry out the program or the policy. There is a limit to the extent to which any single 
capacity can contribute to the overall capacity of a system without depending on, and building on the 
other capacities. 

The notion of alignment offers some insight into consideration of the overall capacity of a system. Each 
kind of capacity represents increased ability of the system to achieve its goals but, as mentioned above, 
when a capacity operates in the absence of others, the ultimate benefits are limited. In theory, when 
capacities are aligned, they can bring benefits to the system which far exceed the total of benefits they 
might bring separately — they create the opportunity for a continuous learning organization to grow and 
develop. One might call this collective view of capacity the ’’fitness" of the system for change. 

The field of computer science offers a different but compatible interpretation of capacity which is referred 
to as "performance evaluation". Alexander Voss suggests that performance can be defined "as a sum of 
properties of a system that make it fit for a specific purpose." He refers to the individual properties as 
"performance indices" with performance the weighted sum of these indices. In this model, Voss explains, 
performance is "limited by the weakest link in the chain, in computer science termed a bottleneck..." 
Similarly, the whole capacity of a system can be limited by an insufficient capacity in any single area 
(Voss, 1998). 
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Organizing for Evaluating Capacity in Systemic Reform 

Because there are so many different aspects of capacity, evaluators may find a general organizing 
structure for looking at capacity of use before they set out to identify specific questions and indicators for 
evaluating capacity. Consider a matrix which plots what might be called "subsystems" of the system 
against the different kinds of capacities: 
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Using this matrix, evaluators can focus on the different types of capacity (human, organizational, 
structural and material) as "drivers" of targeted studies designed to match the more immediate strategies 
and goals of the reform. Each kind of capacity has many facets; evaluators need to identify and focus on 
the most significant ones in order to direct their resources most appropriately. Below are some examples 
of ways to target studies organized by types of capacity. 

Human Capacity 

One of the complicating factors in looking at human capacity is the wide range of people involved in a 
systemic effort. Evaluators need to know what central office administrators with capacity look like; what 
a science or mathematics supervisor with capacity looks like; and what a teacher or a principal with 
capacity looks like. Such capacity could take many forms including: access to other professionals, the 
ability to be self-reflective, the ability to facilitate thoughtful, reflective work in others, and knowledge of 
the discipline and of the pedagogical strategies desired in the reform. Evaluators will need to use a range 
of methodologies to systematically identify these qualities in the individuals participating in each 
subsystem of the system. 
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Goertz and others offer one framework for organizing this task. This framework identifies four 
components of the capacity of teachers (and applies primarily to the instructional subsystem), though it 
can apply to any of the individuals in each of the subsystems. The first two components are "knowledge 
and skills/' and include understanding of and expertise in subject matter, pedagogy, and instructional 
practice. Their third component is "will" which others have referred to as "disposition" or inclination 
(Spillane & Thompson, 1997). Will includes attitudes toward a discipline, toward students, toward 
student achievement, and the commitment to student learning (Goertz, et al., 1995). Additionally, will 
includes the attitude individuals have toward change and reform. The fourth component is "views of self" 
which refers to individuals’ beliefs about their roles in reform activities, and the persona they adopt in the 
classroom or the office (Goertz, et al 1995). In a reform that requires leaders and participants to be 
learners themselves, this view of self is critical. These four dimensions are interdependent and interactive 
in that changes along one dimension can produce unexpected changes in another (Goertz et al., 1995). 

Organizational Capacity 

Assessment of organizational capacity also can have many dimensions. Evaluators need to consider 
organizational substructures in a system, the organizational structure of the system as a whole, the 
system's interactions with other organizational units, and their abilities to all work together. One general 
way to organize a study of organizational capacity is to divide it into two general areas: internal 
relationships and external relationships. 

Internal relationships refer to the interactions between various individuals within an organization. In a 
school district, for example, they include the relationships the teachers have with one another, the 
relationships between the teachers and principals, the relationships between the principals and the central 
office, and so on. An organization with high capacity will have relationships that include open, regular 
communication, access to one another, opportunities for collaboration, common language, and shared 
vision and goals. Evaluators can use a range of strategies to gather data in these areas including 
interview, survey and observation. 
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External relationships refer to those relationships between individuals within an organizational unit of an 
education system (such as a district or a state) and individuals outside that organization. Outside 
individuals might include parents, local businesses, community organizations, and institutions of higher 
education. Entities such as these can sometimes be an important source of expertise and support that 
contribute to the system's current and growing capacity. However, evaluators must look at more than the 
system's ability to communicate with these entities; they also must ask questions which focus on their 
knowledge, skills and expertise for if they themselves have little capacity, they are not likely to help the 
system. 

The literature offers other ways to think about organizational capacity. O'Day and others, for example, 
discuss organizational capacity in five areas: 1) Vision and Leadership; 2) Collective Commitment and 
Cultural Norms (to realize the vision); 3) Knowledge or Access to Knowledge; 4) Organizational 
Structures and Management (conducive to learning and improvement); and 5) Resources (O'Day et al., 
1995). Although the classification scheme in this paper groups some of these areas with other capacities, 
evaluators may wish to develop composite definitions of capacity that are most meaningful to them. The 
classification itself isn't of the most importance; the primary goal is to organize a study of capacity which 
will help the evaluator come to some meaningful conclusions. 

Structural Capacity 

Evaluators may wish to focus a study of "Structural Capacity” by dividing it into two categories: content 
and process. Content related elements are those that reflect the philosophy and substance of the reform 
including curriculum, assessments, professional development programs, and existing and emerging 
policies that serve as scaffolds for change (teacher certification policies, hiring policies, etc.). Process 
elements are those structures that allow for the change to happen, but are somewhat content-free 
including: scheduled time for reflecting on changes and considering improvements, processes for making 
adjustments to the school improvement programs based on feedback from students and schools, and 
forums for open communication among and between administrative offices and schools. 
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Material Capacity 

Financial resources could be assessed by looking at various documents including budgets, descriptions of 
allocation of funding sources, grant applications and use of grant moneys, and by studying the 
organization and use of materials. 



Of course there are other ways to use the matrix. For example, evaluators could move horizontally across 
the matrix, looking at the human, organizational, structural and material capacities of each subsystem. 
With this approach, looking at each cell, the evaluator must consider both, the systemic reform effort , and 
the system itself This is the best way to ensure that judgments about the likelihood of continuous growth 
are accurate, as the leaders and primary participants of the reform may not necessarily represent the 
leadership and population of the system as a whole (at least not at first). Even though reform leaders may 
have high capacity to manage the reform and even the skills to manage the larger system, those who are 
actually in system positions of leadership may not be as capable. 
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Challenges for Identifying and Assessing Capacity in Systemic Reform 

Finally, even after an evaluator has an organizing structure that works, there are other challenges to 
consider. For example, even though capacity may exist or be developing, one may not necessarily be able 
to see it at the time of evaluation. A teacher may have increased his/her understanding of high quality 
mathematics pedagogy and instructional materials (increased human capacity) but still not yet have 
changed his/her classroom practice (evidence of that increased capacity). The evaluator will need to 
consider not only what capacity looks like at a given moment but also what capacity looks like when it is 
in the process of growing. 
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A second challenge lies in the fact that although evidence of capacity may be present at any given time in 
a reform, that evidence may not necessarily demonstrate whether that capacity will continue to grow. For 
example, a community or state may have a policy that mandates a high-quality standards-based 
curriculum (evidence of current structural capacity). That, however, is quite different than having a policy 
that creates the conditions for a curriculum selection process that will allow adoption of standards-based 
curriculum well into the future (evidence that capacity may continue to grow). Practices that seem 
effective for the moment may not contribute to the growth of capacity in the future. 

And finally, capacity may be judged in many ways, using many different questions or indicators. 
Therefore, there is no single scale by which capacity can be measured. Evaluators need to decide whether 
to judge capacity in light of the goals of the reform, by a theoretical ideal, or on the change in capacity 
over time. 

Facing these and other challenges, evaluation of capacity within the context of systemic reform is a 
formidable task. But, it also is an exciting one. With the turn in the evolution of educational change theory 
to include systemic reform also comes an opportunity for educational evaluators to explore new ideas, 
tools, strategies and insights to bring to the educational change process. Educational evaluators and 
researchers are up to this challenge; we simply need to recognize that we have the capacity to look at our 
work and the work of systemic reform with the new, innovative eyes it requires. 
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